Operational Transparency Study
Testing the impacts of operationally transparent communication approaches on COVID-19 vaccine intentions among unvaccinated Canadians
Impact Canada, Impact & Innovation Unit, Privy Council Office
Summary
In collaboration with Drs. Michael Norton and Ryan Buell from Harvard Business School, Impact Canada conducted an online survey experiment to test the impacts of operationally transparent communication approaches on COVID-19 vaccine intentions among unvaccinated Canadians. Operational transparency refers to the open communication of “behind-the-scenes” work that an organization undertakes through its operating processes (Buell and Norton, 2011) - in the present case, communication about the COVID-19 vaccine development and approval process in Canada. This experiment was conducted between August 19 and October 4, 2021 with a sample of ~1,550 unvaccinated Canadians. The study provided rich insights into the intentions, attitudes, and behaviours of Canadians who had not received a single dose of any COVID-19 vaccine at a time when a large majority (80%+) of Canadians had received at least one dose. The results of the experiment suggest that messaging employing principles of operational transparency - specifically those describing the funneling of vaccine candidates through the development process - is more effective at boosting self-reported vaccine intentions among unvaccinated Canadians, compared to standard Government of Canada communications approaches.
Background Research and Origins
In March 2020, the IIU launched a program of applied behavioural science research to support the Government’s COVID-19 response efforts in accurately and effectively promoting key behaviours recommended by public health experts. As one part of this research, the IIU - in close partnership with the Public Health Agency of Canada (PHAC) - studied Canadians’ attitudes and intentions related to COVID-19 vaccines.
Using online survey and experimentation platforms, the IIU has conducted ‘deep-dive’ explorations of critical public health behaviours and tested public health messaging using experimental and quasi-experimental designs. Studies to date have collected data from tens of thousands of Canadians - identifying factors such as best-performing messaging strategies and policy intervention opportunities. These deep-dive explorations have included multiple studies examining the impacts of various behaviourally-informed, communication-based interventions on vaccine acceptance in the COVID-19 context, including three studies that applied principles of operational transparency.
Operational transparency refers to the disclosure of “behind-the-scenes” work that an organization undertakes through its operating processes (Buell and Norton, 2011). While the IIU has tested the effects of operational transparency in the past, previous work focused primarily on written-format interventions and explored the application of operational transparency narrowly as one condition within each larger study.
IIU studies, in addition to those in the literature, provided preliminary evidence on the value of operational transparency into the systems that develop, test and approve vaccines, as a means of improving understanding, trust, and confidence in vaccines. Building on this preliminary evidence, the IIU conducted a dedicated experiment to better understand the effects of different forms of social media-based operationally transparent messaging on COVID-19 vaccine acceptance and uptake, testing 4 visual instantiations aimed at:
- Disclosing the ‘end-to-end’ process for vaccine development, testing, authorization and regulation in Canada; and
- Describing and visualizing the concept of benefits outweighing risks in the context of vaccine authorization.
Study Design
A sample of 1,554 unvaccinated Canadians were recruited between August 19 and October 4, 2021 to participate in an online survey experiment deployed through Qualtrics. Efforts were made to ensure that the sample reflected the population distribution in Canada(using the 2016 Canadian Census); however quota-matching was relaxed due to the difficulty of recruiting a sufficient number of unvaccinated Canadians, who were a minority in the population at the time of data collection.
Respondents were screened into the study based on their self-reported vaccination status. Only respondents with no doses of a COVID-19 vaccine were selected to complete the study.
Figure 1 - Sample Selection
Text version - Figure 1
This figure illustrates the number of participants in the final analyses. Initially, 12,542 participants were recruited through the Qualtrics survey platform (not including the 455 people identified as spam participants). 10,311 participants consented to participate (540 participants did not consent, and 1690¬ participants did not advance past the consent screen). Of these participants, 2,078 indicated that they had not received any COVID-19 vaccines; 1,794 participants completed the study, and of those 1,554 were included for analysis as the remaining participants were excluded for demonstrating bot characteristics, or taking too long to complete the survey.
Screened-in respondents were randomly assigned to one of six groups (two control conditions and four operationally transparent intervention conditions; see below). Stratified randomization was applied to ensure the balance of the treatment/control groups with respect to age and gender. Each group viewed a different experimental condition before being directed to a survey that queried vaccine intentions1, among a suite of other self-reported COVID-related beliefs, attitudes, and behaviours.
Figure 2 - Study Design
Text version - Figure 2
The graphic illustrates how participants were assigned to conditions, and the visual they saw as a result. All participants were assigned to one of six conditions. Those in the Passive Condition saw no visuals; those in the Active Control saw status-quo messaging related to COVID-19 vaccine safety; those in Intervention #1 saw visuals relating to the end-to-end process of vaccine development; those in Intervention #2 saw a visual that compared the typical vaccine development process with the one used for the COVID-19 vaccine; those in Intervention #3 saw the Success Funnel condition detailing how many vaccine candidates were at each stage of the development process; and those in Intervention #4 saw visual detailing the phrase “benefits outweigh the risks”. Following this, all participants were asked to complete measures related to the main outcome variables (intentions to get the COVID-19 vaccine, trust/confidence in COVID-19 vaccines); there behaviour while viewing the conditions was also collected. Finally, all participants were also given additional questions that measured other variables of interest (E.g. perceptions/knowledge of vaccine development, testing and the regulation process; beliefs/behaviour related to COVID-19; demographic characteristics; etc.)
Conditions
The experimental conditions were mock Twitter threads, each consisting of five tweets. Participants were presented with the first tweet of a thread and were able to expand the thread, and click on a link within the thread that went to a page with more information on vaccine approvals. The experimental conditions were as follows:
1. Passive Control (No Visual)
Respondents did not see a visual condition and were directed right to the survey.
2. Active Control (Standard Government of Canada Messaging)
Displayed a series of real-world tweets about vaccine safety that were previously posted on HC/PHAC social media channels
View mock Twitter thread
(1/5) Fact: Only vaccines that are proven to be safe, effective, and of high quality are authorized for use in Canada. When your turn comes, roll up your sleeve and get the CovidVaccine.
(2/5) Fact: Vaccines are critical in our efforts to stop #Covid19. Vaccines are safe, effective, and work.
(3/5) Fact: Vaccines are our number one protection against #COVID19. #VaccinesWork
(4/5) Fact: The #CovidVaccine can’t give you the virus – it protects you against it. #VaccinesWork
(5/5) Fact: #COVID19 vaccines are safe and effective and can’t change your DNA. #VaccinesWork
3. Treatment 1 (End-to-End Process)
Visualized the main stages in the vaccine development and approval process in Canada, aiming to address concerns that not enough testing and research has been done.
View mock Twitter thread
(1/5) Any #COVID-19 vaccines approved in Canada must pass a rigorous assessment and approval process. View this thread to learn about each step. #SleevesUp
(2/5) Scientists run studies to find vaccines that could help us develop immunity against COVID-19. Vaccines are tested in the laboratory and with animals to make sure they are safe before testing in humans.
(3/5) Vaccines are tested in three phases of clinical trials, starting with a small number of volunteers and expanding to tens of thousands of volunteers.
(4/5) Health Canada experts complete a detailed review of the scientific data before making a decision to approve or reject the vaccine.
(5/5) After approval, federal, provincial/territorial, and local health authorities continuously monitor vaccines for safety. Canada’s robust system is able to detect adverse events like serious side effects that are too rare to detect in large clinical trials.
4. Treatment 2 (Typical vs. ‘Rapid’ COVID Process)
Visualized the main stages of the vaccine development and approval process in Canada, aiming to address the concern that the COVID-19 vaccine process was rushed. The tweets showed that, while some stages overlapped, no steps were skipped.
View mock Twitter thread
(1/5) A global effort, large investments in research and manufacturing, and new approaches paved the way for the speedy development of #COVID19 vaccines without compromising safety. #SleevesUp
(2/5) The genetic code of the SARS-CoV-2 virus was published in January 2020, allowing scientists from across the globe to carry out laboratory and animal studies to find vaccines that could be safe and effective in humans.
(3/5) Vaccines are tested in three phases of clinical trials, starting with dozens of volunteers and expanding to tens of thousands. Due to high COVID-19 case numbers in many places, and overlapping clinical trial stages, vaccine efficacy was measured quicker than usual.
(4/5) Health Canada experts complete a detailed review of the scientific data before making a decision to approve or reject the vaccine. For COVID-19 vaccines, study data was submitted as it became available, allowing the review process to start much earlier.
(5/5) After approval, federal, provincial/territorial, and local health authorities continuously monitor vaccines for safety. Canada’s robust system is able to detect adverse events like serious side effects that are too rare to detect in large clinical trials.
5. Treatment 3 (Success Funnel)
Described the number of vaccine candidates at each stage of the vaccine development and approval process with a visual of climbing a mountain and reaching the “peak” - approval. This condition aimed to show that only vaccines that meet safety and effectiveness criteria make it to the approval stage, and some candidates fall out at each stage of the process.
View mock Twitter thread
(1/5) Any #COVID-19 vaccines approved in Canada must pass a rigorous assessment and approval process.View this thread to learn about each step. #SleevesUp
(2/5) Scientists run studies to find vaccines that could help us develop immunity against COVID-19. Vaccines are tested in the laboratory and with animals to make sure they are safe before testing in humans.
(3/5) Vaccines are tested in three phases of clinical trials, starting with a small number of volunteers and expanding to tens of thousands of volunteers.
(4/5) Health Canada experts complete a detailed review of the scientific data before making a decision to approve or reject the vaccine.
(5/5) After approval, federal, provincial/territorial, and local health authorities continuously monitor vaccines for safety. Canada’s robust system is able to detect adverse events like serious side effects that are too rare to detect in large clinical trials.
6. Treatment 4 (Benefits Outweigh the Risks)
Explained the meaning behind “the benefits outweigh the risks” for vaccines in Canada. The tweets quantified the benefits (vaccine efficacy) and risks (side effects) to help people understand what goes into risk-benefit calculations during the vaccine review and authorization process.
View mock Twitter thread
(1/5) For all COVID-19 vaccines used in Canada, the benefits of the vaccine must outweigh the risks. But what type of “benefits” and “risks” are we talking about when making these decisions? #SleevesUp
(2/5) The COVID-19 virus, and its variants, can infect many people very quickly and the estimated mortality rate is much higher than the flu. Older adults and those with underlying conditions are at higher risk for poor outcomes.
(3/5) All COVID-19 vaccines approved for use in Canada are very effective in preventing COVID-19. Studies suggest that vaccinated people are 98% less likely to be hospitalized.
(4/5) All COVID-19 vaccines have minimal reported side effects (like headaches, feeling feverish, fatigue) and serious side effects are extremely rare (1 in 18,310 people)
(5/5) As COVID-19 infections grow or flatten, the risk of the virus changes and so may recommendations. Health Canada is constantly ensuring that the benefits of vaccines outweigh their risks and the risks posed by COVID-19.
Study Findings
Finding 1
Status quo approaches to communicating about vaccine safety may have unintended consequences for unvaccinated Canadians.
On average, respondents in the Active Control group (i.e.,tweets that had previously been communicated by the Government of Canada about vaccine safety) reported statistically significantly lower intentions to be vaccinated, compared to respondents who were exposed to the Passive Control (i.e., those who were shown nothing).
Text version - Figure 3
This figure illustrates the difference in average intention to get the COVID-19 vaccine between those in the Passive Control group, and those in the Active Control group. Those in the Passive Control group have significantly higher intentions to vaccinate (M = 2.96) relative to the Active Control group (M = 2.69).
Finding 2
Respondents exposed to the “Success Funnel” condition reported, on average, 13% higher intentions to vaccinate against COVID-19, relative to status quo Government communications about vaccine safety (i.e., Active Control condition).
On average, those in the “Success Funnel” condition had vaccine intentions that were significantly (statistically) higher relative to the active control group2. Vaccine intentions for those in the other three operational transparency conditions were not significantly different from the Active Control condition. Additionally, intentions were not significantly different between the operational transparency conditions.
These findings suggest that “showing” the behind-the-scenes work that goes into ensuring vaccines are safe and effective may be more effective than simply “telling” citizens that this is the case.
Text version - Figure 4
A figure illustrating the difference in average vaccine intention between experimental conditions (excluding the Passive Condition). The Active Condition has the lowest average (2.69); the average vaccine intention of participants in OT groups are as follow:
- End-to-end: 2.90
- Typical vs. Rapid: 2.95
- Success Funnel: 3.04
- Benefits > Risks: 2.88
Those in the Success Funnel condition had significantly higher intentions to vaccinate, relative to those in the Active Control condition. Differences between the Active Control and the other OT conditions were not statistically significant. While the Success Funnel condition also performed better than the other OT conditions, there were no statistically significant differences between any of the OT conditions.
Limitations and Future Directions
When interpreting the results of this study, there are some key study limitations to bear in mind. Though the messaging conditions used mock tweets, the experiment was conducted in the context of an online survey, and therefore results have limited ecological validity. Testing these interventions using a simulated social media environment, or, better yet, directly on social media platforms where individuals interact with information in day-to-day life, will provide better estimates of real world impacts. Relatedly, the study only analyzed vaccine intentions and not actual vaccination behaviours. Though common, self-report data is a limitation because there are often discrepancies between individuals' stated intentions and their actions.
It is also worth considering that tweet viewing times in this study were relatively low (median = 12.47 seconds, across conditions). In addition, only 8.7% of respondents expanded the tweet threads. Most study participants, therefore, were only exposed to the first tweet in the threads, and for a relatively short time. The degree to which these engagement metrics align with real-world social media behaviour is unclear, though it is encouraging that we observed significant effects of operationally transparent messaging with relatively brief and low-demand engagement by participants. Tweet viewing and engagement did not vary as a function of condition.
More work is necessary to understand (A) how to best encourage engagement with social media messaging campaigns, and (B) how to maximize message effectiveness in the face of low expected engagement. This may indicate that social media campaigns should be ‘front-loaded’ with operationally transparent content to get the message across without requiring active user engagement.
Finally, while those who viewed the “Success Funnel” condition had higher vaccine intentions, relative to the active control, intentions were similar to those in the passive control condition (i.e. participants who saw no visuals). Future work should examine the potentially amplifying effects of operational transparency with trusted messengers outside of government (e.g. community leaders) - especially in those who distrust government as previous research suggests that the messenger can be just as, or more, important than the message itself.
Library and Archives Canada Cataloguing in Publication
Operational Transparency Study
Information contained in this publication or product may be reproduced, in part or in whole, and by any means, for personal or public non-commercial purposes without charge or further permission, unless otherwise specified. Commercial reproduction and distribution are prohibited except with written permission from the Privy Council Office.
For more information, contact:
Impact Canada, Privy Council Office
85 Sparks Street, Room 1000
Ottawa ON Canada K1A 0A3
info@pco-bcp.gc.ca
© His Majesty the King in Right of Canada, 2024.
Cette publication est également disponible en français : Étude sur la transparence opérationnelle
ISBN:
978-0-660-71277-2
CP22-217/2024E-PDF